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(54) Backlight adjustment processing of image using image generation record information 



(57) A decision as to whether to perform backlight 
adjustment processing is executed using both image 
data and image generation record information, and 

Fig. 11 



when it is decided to execute the same, backlight ad- 
justment processing to increase brightness value of at 
least some pixels in the image data is executed. 
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Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 

[0001] The present invention relates to an image data adjusting technique for adjusting image quality of image data. 
Description of the Related Art 

[0002] Quality of image data generated by a digital still cameras (DSC) or digital video cameras (DVC) can be ad- 
justed in any desired manner using an image retouching application on a personal computer. Image retouching appli- 
cations typically have an image adjusting function that automatically adjusts the quality of image data, and quality of 
image data output from an output device may be improved using this image adjusting function. Known image output 
devices include, for example, CRTs, LCDs, printers, projectors, and television receivers. 

[0003] A printer driver for a printer also has a function for automatically adjusting image quality, and quality of printed 
images can be improved using such a printer driver as well. 

[0004] Brightness is one important factor determining image quality of image data. If the brightness of image data 
has been set to the proper brightness level, the user will recognize the high image quality of the image. Image data 
brightness is strongly affected by the position of the light source when image data is generated. For example, image 
data is sometimes generated under condition in which the sun or other light source is situated behind a subject (back- 
lighting). Under such backlit conditions, in some instances sufficient light will not fall on the side of an intended subject 
that is facing the camera, resulting in generation of image data in which the intended is darker than the proper brightness 
level. Thus, methods that analyze image data brightness and adjust the image data on the basis of the results so as 
to increase brightness in dark areas and improved image quality are used. 

[0005] With the automatic image quality adjusting function provided by an image retouching application or printer 
driver, quality corrections are made on the basis of image data having standard quality characteristics. On the other 
hand, image data being subjected to image processing can be produced under various conditions. For example, the 
position of a subject within an image may be set arbitrarily according to user preference, and shooting location. In 
some instances image data is generated using a flash or other supplemental light source; and shooting takes place 
under different conditions depending on shooting location, for example, shooting outdoors, and shooting indoors. 
[0006] For this reason, in some instances image quality of an image overall cannot be improved through quality 
corrections to an image containing dark areas, made on the basis of image data having standard quality characteristics. 
This problem Is not limited to DSCs, and is a problem common to DVCs and other such image producing devices. 

SUMMARY OF THE INVENTION 

[0007] Accordingly, an object of the present invention to automatically adjust image quality with reference to an 
individual set of image data. 

[0008] In order to attain at least part of the above and related object of the present invention, there is provided an 
output device for outputting an image using image data generated by an image generating device, and image generation 
record information that is associated with that image data and that includes operation information for that image gen- 
erating device at the time of image generation. The output device comprises an image processing device and an image 
output section. The image processing device comprises a judging section , and an image quality adjuster. The judging 
section uses both the image generation record information and the image data in order to perform a backlight decision 
as to whether to execute backlight adjustment processing. The image quality adjuster executes backlight adjustment 
processing to increase brightness value of at least some pixels in the image data when it is decided to execute that 
backlight adjustment processing. The image output section outputs an image according to the image quality-adjusted 
image data. 

[0009] The output device according to the present invention can appropriately make a decision as to whether to 
execute backlight adjustment processing based on both image generation record information and image data. Backlight 
adjustment processing will improve brightness in dark areas having low brightness value. 

[0010] This invention may be embodied in various ways, for example, an image output method and image output 
device; an image processing method and image processing device; a computer program for realizing the functions of 
such a method or device; a storage medium having such a computer program stored thereon; and a data signal em- 
bedded in a carrier wave containing such a computer program. 

[0011] These and other objects, features, aspects, and advantages of the present invention will become more ap- 
parent from the following detailed description of the preferred embodiments with the accompanying drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[0012] Fig. 1 is a block diagram of arrangement of an image output system as an embodiment of the invention. 

[0013] Fig. 2 is a block diagram showing a simplified arrangement of a digital still camera 12. 

5 [0014] Fig. 3 is a conceptual illustration of an example of an arrangement within an image file. 

[0015] Fig. 4 illustrate an example of data structure of an ancillary information storage field 103. 

[0016] Fig. 5 illustrates an example of data structure in an Exif data field. 

[0017] Fig. 6 is a block diagram showing a simplified arrangement of a printer 20. 

[0018] Fig. 7 is a block diagram showing the arrangement of printer 20. 

10 [0019] Fig. 8 is a flowchart showing flow of an image file GF generation process. 

[0020] Fig. 9 is a flow chart showing a processing routine for image processing. 

[0021] Fig. 10 is a flow chart showing a processing routine for image processing. 

[0022] Fig. 11 is a flow chart showing a processing routine for image quality adjustment processing. 

[0023] Figs. 12(a) and 12(b) illustrate the first embodiment of a backlight decision process. 

15 [0024] Figs. 13(a) and 13(b) illustrate the first embodiment of backlight adjustment processing. 

[0025] Figs. 14(a) and 14(b) illustrate in the second embodiment of a backlight decision process. 

[0026] Figs. 15(a) and 15(b) illustrate the third embodiment of a backlight decision process. 

[0027] Figs. 1 6(a) and 1 6(b) illustrate the fourth embodiment of a backlight decision process. 

[0028] Figs. 17(a)-17(c) illustrate a weight W distribution for use in calculating average brightness value in the fifth 
20 embodiment of a backlight decision process. 

[0029] Figs. 18(a) and 18(b) illustrate the fifth embodiment of a backlight decision process. 

[0030] Figs. 19(a) and 19(b) illustrate the sixth embodiment of a backlight decision process. 

[0031] Figs. 20(a)-20(c) illustrate the second embodiment of backlight adjustment processing. 

[0032] Figs, 21 (a)-21 (c) illustrate the third embodiment of backlight adjustment processing. 

25 [0033] Fig. 22 illustrates an example of an image output system. 

[0034] Fig. 23 is a flow chart showing an image processing routine. 

[0035] Fig. 24 is a flow chart showing another example of an image processing routine. 

[0036] Fig. 25 is a flow chart showing another example of an image processing routine. 

30 DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0037] The embodiments of the present invention are described hereinbelow through certain specific preferred em- 
bodiments, in the following order. 

35 A. Arrangement of Image Output System 

B. Arrangement of Image File 

C. Arrangement of Image Output Device 

D. Image Processing in Digital Still Camera 

E. Image Processing in Printer 

40 R Embodiments of Automatic Image quality Adjustment Process 

G. Other Embodiments of Backlight Decision Process 

H. Other Embodiments of Backlight adjustment processing 

I. Arrangement of Image Output System Employing Image Data Processing Device 
J. Variant Examples 
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A. Arrangement of Image Output System: 



[0038] Fig. 1 is an illustration showing an example of an image output system in which may be implemented an 
output device as an embodiment of the invention. Image output system 10 comprises a digital still camera 12 as an 

50 image generating device for generating image files, and a printer 20 as an image output device. Image files generated 
by digital still camera 12 are transferred to printer 20 via a cable CV, or by directly inserting into printer 20 a memory 
card MC on which image files are stored. Printer 20 executes image quality adjustment processing of image data based 
on read out image files, and outputs an image. As the output device, a CRT display, LCD display, or other monitor 14, 
or a projector could be used in addition to or instead of printer 20. The following description is based on the use of a 

55 printer 20 comprising an image quality adjuster and an image-output section, with a memory card MC being inserted 
directly into the printer 20. 

[0039] Fig. 2 is a block diagram showing a simplified arrangement of a digital still camera 1 2. The digital still camera 
12 of this embodiment comprises an optical circuit 121 for collecting optical information; an image acquisition circuit 
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122 for controlling the optical circuit to acquire an image; an image processing circuit 123 for processing acquired 
digital images; a flash 130 serving as a supplemental light source; and a control circuit 124 for controlling the various 
circuits. Control circuit 124 comprises memory, not shown. Optical circuit 121 comprises a lens 125 for collecting optical 
information; an aperture 129 for adjusting the amount of light; and a CCD 128 for converting optical data passing 
5 through the lens into image data. 

[0040] Digital still camera 12 stores acquired images on a memory card MC. The typical storage format of image 
data in digital still camera 12 is the JPEG format, but other storage format, such as TIFF format, GIF format, BMP 
format, or RAW data format, could be used. 

[0041] Digital still camera 1 2 further comprises a Select/Set button 1 26 for setting various shooting parameters, and 
10 a liquid crystal display 127. Liquid crystal display 127 is used to preview photographed images, and when setting the 
aperture or other parameters using the Select/Set button 126. 

[0042] When a photograph has been taken with a digital still camera 12, image data and image generation record 
information are stored as an image file in memory card MC. Image generation record information can include aperture 
value and other parameter settings at the time of shooting (time of generation of image data) . 
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B. Arrangement of Image File: 



[0043] Fig. 3 is a conceptual illustration of an example of the structure of an image file that can be used in this 
embodiment. Image file GF comprises an image data storage field 101 for storing image data GD; and an image 

20 generation record information storage field 1 02 for storing image generation record information Gl. Image data GD is 
stored, for example, in JPEG format, while image generation record information Gl is stored, for example, in TIFF 
format (a format in which data and data fields are specified using tags). The terms file structure and data structure in 
this embodiment refer to file or data structure in the form in which a file or data etc. is stored in a memory device. 
[0044] Image generation record information Gl is produced at the time when the associated image data is generated 

25 or shot by digital still camera 12 or other such image generating device, and includes the following settings. 

light source (type of light source) 
flash (Flash On/Off) 
subject distance 
30 . subject distance range 
subject area 
flash intensity 
aperture value 
ISO speed rating 
35 shooting mode (scene mode) 

maker name 
model name 
gamma value 

40 [0045] The image file GF in this embodiment may basically comprise the aforementioned image data storage field 
101 and image generation record information storage field 102; or may have a file structure according to an existing 
standardized file format. The following specific description pertains to the case where the image file GF pertaining to 
this embodiment conforms to the Exit file format. 

[0046] An Exit file has a file structure in accordance with the digital still cameran image file format specification (Exif); 

45 the specification has been proposed by the Japan Electronics and Information Technology Industries Association (JEI- 
TA). Similar to the conceptual diagram in Fig. 3, the Exif file format comprises a JPEG image data storage field for 
storing image data in the JPEG format; and an ancillary information storage field for storing information of various kinds 
relating to stored JPEG image data. The JPEG image data storage field corresponds to the image data storage field 
101 in Fig. 3, and the ancillary information storage field to the image generation record information storage field 102. 

50 The ancillary information storage field stores image generation record information relating to a JPEG image, such as 
shooting date/time, aperture value, and subject distance. 

[0047] Fig. 4 illustrates an example of data structure of ancillary information storage field 1 03. In the Exif file format, 
hierarchical tags are used to designate data fields. Each data field can contain a plurality of subordinate data fields 
identified by subordinate tags. In Fig. 4, areas enclosed by rectangles represent single data fields, with tag names 
55 noted at upper left. In this embodiment, three data fields whose tag names are APP0, APP1 , and APP6 are contained. 
The APP1 data field contains within it two data fields whose tag names are IFDO and IFD1 . The IFD0 data field contains 
three data fields whose tag names are PM, Exif, and GPS. Data and data fields are stored according to a prescribed 
address or offset value; address or offset value can be searched by means of tag name. On the output device, data 



4 



3NSDOCID: <EP 14041 18A1_I_> 



EP1 404 118 A1 



corresponding to desired information can be acquired by means of specifying and address or offset value corresponding 
to the desired information. 

[0048] Fig. 5 illustrates an example of data structure (data tag names and parameter values) in the Exit data field in 
Fig. 4, wherein tag name can be referenced by tracing in the order APP1-IFD0-Exif. As shown in Fig. 4, the Exif data 
5 field can include a data field whose tag name is MakerNote; the MakerNote data field can in turn include a plurality of 
items of data, although these are omitted in Fig. 5. 

[0049] As shown in Fig. 5 S the Exif data field stores parameter values relating to information such as light source, 
flash, subject distance, subject area, flash intensity, aperture value, and ISO speed rating. In this embodiment, light 
source is used as information relating to the location of the subject of the image data, such as indoors or outdoors; 
10 flash is used as flash information for a supplemental light source; subject distance is used as information relating to 
the distance between a subject of image data and the image generating device; and subject area is used as subject 
position information, respectively. 

[0050] Light source information indicates the type of light source at the time of generation of image data, and may 
be selected from among, for example, daylight, clear sky, cloudy, shade, fluorescent, and tungsten. Of these settings, 
15 a decision that the subject is situated outdoors may be made when the device is set to daylight, clear sky, cloudy, or 
shade. A decision that the subject is situated indoors may be made when the device is set to fluorescent or tungsten. 
[0051] Flash information indicates operation of the flash, and can include four sets of information relating to operating 
modes and operation result thereof. Operating mode may be set from among a plurality of values including the following 
three values, for example. 

20 

1 : forced flash mode 
2: flash off mode 
3: auto flash mode 



25 [0052] Operation result may be set from among two values, Flash on or Flash off, for example. The decision as to 
whether or not there was illumination by the supplemental light source during generation of image data may be carried 
out using this operation result. 

[0053] Certain image generating devices are equipped with a mechanism for sensing reflected light from a subject 
illuminated with light from a flash. In the event that a flash cover or other obstacle blocks the flash light, or the flash 

30 does not fire despite being operated, no light illuminates the subject. Such instances can be identified through the 
presence or absence of reflected light. Flash information can include information relating to the presence/absence of 
a reflected light sensing mechanism, and to the presence/absence of reflected light sensed at the time the image data 
is generated (or time of shooting). In the event that a reflected light sensing mechanism is present but sensed reflected 
light is absent, it can be decided that there is no illumination by the supplemental light source, even if the aforementioned 

35 operation result is Flash on. 

[0054] In the event that an image generating device is not equipped with a flash, the flash information may be set to 
"Flash off. Where set to "Flash off', it can be decided that there has been no illumination by a supplemental light source. 
[0055] Subject distance information indicates the distance between the image generating device and a subject at 
the time of generation of image data. For example, it can be set in meter units, based on distance information set to 

40 match the focal point at the time that image data is generated. 

[0056] Subject area information indicates the position of a subject in an image, and center coordinates in the image 
are set as the parameter value. Where a circular or rectangular area is set to indicate the size of the subject, the radius 
of the circle or width of the rectangle can be set in association therewith. 

[0057] Flash intensity information indicates the quantity of light emitted by the flash at the time that image data is 
45 generated; the unit of measurement thereof is BCPS (Beam Candle Power Seconds), for example. 

[0058] Aperture value is of the value at the time that image data is generated; F number is used as the parameter 
value. Thus, a larger aperture value means a smaller aperture. 

[0059] ISO speed rating information indicates the sensitivity of the optical circuit at the time that image data is gen- 
erated, and is set to a parameter value equivalent to ISO speed which is an index of sensitivity of silver salt film. ISO 
so speed is used in combination with other parameters relating to image generation, such as aperture value, to set ap- 
propriate image generating parameters or shooting parameters. In an image generating device such as a digital still 
camera, by using the corresponding ISO speed as an indicator of the sensitivity of the optical circuit, image generating 
parameters such as aperture value can be set easily 

[0060] The above information is all operating information for the image generating device. Operating information 
55 may be set by the user in association with generation of image data, or set automatically by the image generating 
device. Certain image generating devices permit the user to set a shooting mode or a scene mode, with the image 
generating device then automatically setting related parameters (aperture value, ISO speed etc.) with reference to the 
selected shooting mode. Shooting mode can be selected from among a number of predetermined modes, for example, 



5 



iNSDCCID: <EP 14041 18A1_L> 



EP1 404 118 A1 



standard mode, portrait mode, landscape mode, and night scene mode. Where standard mode has been selected as 
the shooting mode, parameters relating to image generation are set to standard values. 

[0061] Information associated with image data is also stored appropriately in fields other than the Exit data field in 
Fig. 4. For example, maker name and model name are stored in the data field whose tag name is IFDO, as information 
identifying the image generating device. 

C. Arrangement of Image Output Device: 

[0062] Fig. 6 is a block diagram showing a simplified arrangement of printer 20 in the present embodiment. Printer 
20 is a printer capable of image output, for example, an ink jet printer that ejects ink of four colors, cyan C, magenta 
M, yellow Y, and black K, on a print medium to reproduce an image. An electrophotographic printer that transfers and 
fixes toner onto a print medium may also be used. In addition to the four colors indicated above, light cyan LC which 
is lighter in density than cyan C, light magenta LM which is lighter in density than magenta M, dark yellow DY which 
is darker in density than yellow Y may be used as ink. Where monochromatic printing is performed, the arrangement 
may employ black K only; or red R or green G may be used. The type of ink or toner used can be selected depending 
on the characteristics of the image for output. 

[0063] As shown in the drawing, printer20 comprises a mechanism for driving a print head 21 1 mounted on a carriage 
21 , to eject ink and form dots; a mechanism for reciprocating carriage 21 in the axial direction of a platen 23 by means 
of a carriage motor 22; a mechanism for feeding printer paper P by means of a paper feed motor 24; and a control 
circuit 30. By means of these mechanisms, printer 20 functions as an image output section. The mechanism for recip- 
rocating carriage 21 in the axial direction of platen 23 is composed of a slide rail 25 extending parallel to the axis of 
platen 23, for slidably retaining carriage 21 ; a pulley 27 coupled via an endless drive belt 26 to a carriage motor 22; 
and a position sensor 28 for sensing the home position of carriage 21 . The mechanism for feeding printer paper P is 
composed of platen 23; paper feed motor 24 which rotates platen 23; an auxiliary paper feed roller, not shown in the 
drawing; and a gear train (not shown) for transmitting rotation of paper feed motor 24 to platen 23 and the auxiliary 
paper feed roller. 

[0064] Control circuit 30 exchanges signals with a printer control panel 29 while appropriately controlling operation 
of paper feed motor 24, carriage motor 22, and print head 211 . Printer paper P supplied to printer 20 is set pinched 
between platen 23 and the auxiliary paper feed roller, and advanced by a predetermined amount depending on the 
rotation angle of platen 23. 

[0065] Carriage 21 has a print head 211 , and is designed for installation of an ink jet cartridge of utilizable ink. On 
the bottom face of print head 211 are disposed nozzles for ejecting utilizable ink (not shown). 

[0066] Fig. 7 is a block diagram showing an arrangement of printer 20, centered on the control circuit 30 of printer 
20. Within control circuit 30 are disposed a CPU 31 , PROM 32, RAM 33, a memory card slot 34 for acquiring data from 
a memory card MC, a peripheral device input/output section (PIO) 35 for exchanging data with paper feed motor 24, 
carriage motor 22, and a drive buffer 37. Drive buffer 37 is used as a buffer for supplying dot on/off signals to print 
head 21 1 . These components are interconnected by a bus 38, enabling exchange of data among them. Control circuit 
30 is also provided with a transmitter 39 for outputting a drive waveform at predetermined frequency, and a distributed 
output device 40 for distributing the output of transmitter 39 to print head 211 at predetermined timing. 
[0067] Control circuit 30, while synchronizing with operation of paper feed motor 24 and carriage motor 22, outputs 
the dot data to drive buffer 37 at predetermined timing. Control circuit 30 also reads out an image file from memory 
card MC, analyzes the ancillary information, and performs image processing based on the resultant image generation 
record information. That is, control circuit 30 functions as a judging section and image quality adjuster. The flow of 
image processing executed by control circuit 30 is described later in detail. 

D. Image Processing in Digital Still Camera: 

[0068] Fig. 8 is a flowchart showing a process flow for generating an image file GF in digital still camera 12. 
[0069] Control circuit 1 24 (Fig. 2) of digital still camera 1 2 generates image data GD in response to a shoot request, 
for example, depression of a shutter button (Step S100). Where aperture value, ISO speed, shooting mode and other 
parameter settings have been made, image data GD is generated using the set parameter values. 
[0070] Control circuit 124 stores the generated image data GD and image generation record information Gl as an 
image file GF on memory card MC (Step S110), and terminates the processing routine. Image generation record in- 
formation Gl includes parameters used at the time of image generation, such as aperture value, ISO speed, shooting 
mode and/or other arbitrarily set parameter values, and parameter values set automatically, such as maker name, and 
model name. Image data GD is stored in an image file GF after being converted from an RGB color space to a YCbCr 
color space, and compressed in JPEG format. 

[0071] By means of the aforementioned processes executed in digital still camera 12, image generation record in- 
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formation Gl that includes various parameter values at the time of generation of image data is set in the image file GF 
stored on memory card MC, together with the image data GD. 

E. Image Processing in Printer: 

5 

[0072] Fig. 9 is a flowchart showing a processing routine for image processing in printer 20 of the present embodi- 
ment. In the following description, it is assumed that a memory card MC having an image file GF stored thereon is 
inserted directly into printer 20. When memory card MC has been inserted into memory card slot 34, the CPU 31 of 
control circuit 30 (Fig. 7) of printer 20 reads out the image file GF from memory card MC (Step S200). Next, in Step 

w S210, CPU 31 searches in the ancillary information storage field of image file GF for image generation record infor- 
mation Gl indicating information at the time that the image data was generated. In the event that image generation 
record information Gl is found (Step S220: Y), CPU 31 acquires and analyzes the image generation record information 
Gl (Step S230). On the basis of the analyzed image generation record information Gl, CPU 31 executes image process- 
ing, described later (Step S240), outputs the processed image (Step S250), and terminates the processing routine. 

15 [0073] An image file created by a drawing application, on the other hand, will not contain image generation record 
information Gl having information such as aperture value and the like. If CPU 31 cannot find image generation record 
information Gl (Step S200: N), it performs standard processing (Step S260), outputs the processed image (Step S250), 
and terminates the processing routine. 

[0074] Fig. 1 0 is a flowchart showing a processing routine for image processing (corresponding to Step S240 in Fig. 
20 9) based on image generation record information. The CPU 31 of control circuit 30 (Fig. 7) of printer 20 reads out image 
data GD from the read out image file GF (Step S300) 

[0075] As mentioned previously, digital still camera 12 stores image data GD as JPEG format files, and in a JPEG 
format file image data is stored using an YCbCr color space. In Step S310, CPU 31 executes an operation using 3x3 
matrix S to convert image data based on an YCbCr color space into image data based on an RGB color space. This 
25 matrix operation is represented by the following arithmetic expression, for example. 

[Eq. 1] 
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[0076] Where the color space of image data generated by digital still camera 1 2 is wider than a predetermined color 
space, for example, the sRGB color space, image data based on the RGB color space obtained in Step S310 may in 
some instances contain valid data that is outside the defined area of the RGB color space. Where image generation 
record information Gl instructs this out-of-defined area data to be handled as valid data, the out-of -defined area data 
will be kept, and subsequent image processing continues. In the absence of an instruction to handle out-of-defined 
area data as valid data, out-of-defined area data is clipped to the defined area. For example, where the defined area 
is 0 -255, negative value data of less than 0 is rounded to 0, and data above 255 to 255. In the event that the color 
space that can be represented by the image output section is not wider than a predetermined color space, for example, 
the sRGB color space, it is preferable to clip to the defined area, regardless of any instruction in the image generation 
record information Gl. Such instances would include, for example, cases where image is output to a CRT, whose 
available color space is the sRGB color space. 

[0077] Next, in Step S320 : CPU 31 performs gamma correction and an operation employing a matrix M, to convert 
image data based on an RGB color space to image data based on an XYZ color space. Image file GF can contain 
gamma value and color space information at the time of image generation. In the event that image generation record 
information Gl includes this information, CPU 31 acquires the gamma value of the image data from the image generation 
record information Gl, and executes a gamma conversion process of the image data using the acquired gamma value. 
CPU 31 then acquires color space information for the image data from the image generation record information Gl, 
and executes a matrix operation on the image data using a matrix M that corresponds to the color space. In the event 
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that image generation record information Gl does not contain a gamma value, a gamma conversion process can be 
executed using a standard gamma value. A gamma value and matrix for the sRGB color space may be used respectively 
as this standard gamma value and matrix M. The matrix operation may be given by the following arithmetic expression, 
for example. 
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[0078] The color space of image data obtained after the matrix operation has been executed is an XYZ color space. 
The XYZ color space is an absolute color space, and is a device-independent color space not dependent on a device 
such as a digital still camera or printer. Thus, device-independent color matching can be carried out by means of color 
space conversion through the XYZ color space. 

[0079] Next, in Step S330, CPU 31 performs an operation employing a matrix N" 1 , and executes inverse gamma 
correction to convert image data based on the XYZ color space to image data based on the wRGB color space. During 
inverse gamma correction, CPU 31 acquires a printer-side gamma value from PROM 32, and executes inverse gamma 
correction of the image data using the inverse of the acquired gamma value. CPU 31 then acquires from PROM 32 a 
matrix N~ 1 that corresponds to the conversion from the XYZ color space to the wRGB color space, and performs a 
matrix operation on the image data using this matrix N" 1 . This matrix operation may be given by the following arithmetic 
expression, for example. 
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[0080] Next, in Step S340, CPU 31 executes automatic adjustment processing of image quality. Automatic image 
quality adjustment processing in this embodiment involves the use of image generation record information contained 
in the image file GF. Automatic image quality adjustment processing is described later. 

[0081] Next, in Step S350, CPU 31 executes a CMYK color conversion process and a halftone process for the 
25 purpose of printing. In the CMYK color conversion process, CPU 31 refers to a look-up table (LUT), stored in PROM 
32, for conversion from the wRGB color space to the CMYK color space, and converts the color space of the image 
data from the wRGB color space to the CMYK color space. That is, image data consisting of RGB muiti level values 
is converted to image data for use by printer 20, consisting, for example, of mutti level values for six colors, C (Cyan), 
M (Magenta), Y (Yellow), K (Black), LC (Light Cyan), and LM (Light Magenta). 
30 [0082] CPU 31 executes a halftone process to produce halftone image data from the colorrcon verted image data. 
This halftone image data is sequenced in the order of transmission to drive buffer 37 (Fig. 7) to produce the final print 
data, whereupon the processing routine terminates. Image data processed by means of this processing routine is 
output in Step S250 of the image processing routine shown in Fig. 9. 



35 R Embodiment of Automatic Image Quality Adjustment Process: 



[0083] Fig. 11 is a flowchart showing a processing routine for automatic image quality adjustment processing (cor- 
responding to Step S340 in Fig. 10) in this embodiment. CPU 31 (Fig. 7) analyzes the image generation record infor- 
mation Gl and acquires parameter values for flash information etc. (Step S400). Next, in Step S410, CPU 31 , based 

40 on the acquired parameter values, makes a backlight decision as to whether to execute backlight adjustment process- 
ing. If it is determined that there is backlighting, i.e., it is decided to execute backlight adjustment processing (Step 
S410: Y), in Step S420, CPU 31 executes a brightness adjustment process to increase brightness value. 
[0084] After executing the brightness adjustment process, in Step S430, CPU 31 executes a chroma adjustment 
process to increase chroma or saturation. In areas of low brightness value, there is a strong tendency for chroma to 

45 be low. Thus : if only brightness value is increased by means of a brightness adjustment process, formerly dark areas 
may become whitish areas of high brightness value but low chroma. The chroma adjustment process will attain more 
vivid images. 

[0085] After executing the chroma adjustment process, in Step S440, CPU 31 executes a noise elimination process. 
Noise, which does not stand out when brightness value is low, may in some instances stand out when brightness value 
50 is increased. In this embodiment, a noise elimination process is executed in order to prevent noise from standing out. 
The noise elimination process may use various filters including a median filter, and an unsharp mask. 
[0086] The process flow shown in Figs. 9-11 is applicable to other embodiments described later. 

F1 . Embodiment 1 of Backlight Decision Process: 

55 

[0087] Figs. 12(a) and 12(b) illustrate the backlight decision process in this embodiment. IMG12 in Fig. 12(a) shows 
an image under backlit conditions. In image IMG12, a human figure H is the subject, with the sun S located in the 
background. Since the light source, namely, the sun S is situated behind the human figure H, human figure H appears 



9 



JNSDOC1D: <EP 14041 1SA1_I_> 



EP1 404 118 A1 



dark. Within image IMG12 is established a subject area center SAC and a subject area SA; the information for SAC 
and SA is included in the image generation record information. Human figure H is located at the subject area center 
SAC. Subject area SA is set to match the size of human figure H. In this example, subject area SA is rectangular. 
[0088] Fig. 12(b) shows the backlight decision process of this embodiment. In this embodiment, it is judged that 
5 backlighting is present when the following condition is met. 

[0089] Condition (b1): Average brightness value BAave in the subject area is equal to or less than a value obtained 
by multiplying an average brightness value for the entire image by a coefficient k. 

[0090] Specifically, it is judged that backlighting is present when brightness of a subject area is lower than overall 
image brightness. The value of the coefficient k may be determined on the basis of sensory test of image output results. 

10 For example, a value of 0.6 may be used as a predetermined coefficient k. The decision concerning Condition (b1) is 
executed using both subject area information and image data GD. By executing decisions based on subject area 
information in this way, an image in which a subject having a desired location and size is dark may be selected appro- 
priately as a target for backlight adjustment processing. It should be noted that in this decision process the image data 
GD is analyzed while assigning zero weighting to the area outside the subject area. In other words, the image data 

is GD is analyzed using a weight distribution of different magnitude in the subject location than in the area outside the 
subject area. 

F2. Embodiment 1 of Backlight Adjustment Processing: 

20 [0091] Figs. 13(a) and 13(b) illustrate backlight adjustment processing in this embodiment. Fig. 13(a) shows an 
example of a brightness value distribution in a backlit image. In a backlit image, both bright areas and dark areas are 
present. Thus, there are large numbers of pixels having high brightness value and pixels having low brightness value. 
Particularly where the subject is dark due to backlighting, as in Fig. 12(a), the number of pixels having low brightness 
value is substantially large. 

25 [0092] Fig. 13(b) shows the relationship of brightness value input level Yin to brightness value output level Yout in 
backlight adjustment processing (or brightness adjustment processing) of this embodiment. Graph GA is arranged so 
that output level Yout is greater than input level Yin. In particular, the design is such that the increase thereof is greater 
in areas of low input level Yin. Where brightness values are adjusted using this graph GA, brightness can be improved 
in dark areas. Values for increase in brightness value may be predetermined based on sensory test of image output 

30 results. 

[0093] Graph GB, in contrast to graph GA, raises brightness values only in areas of low input level Yin. By adjusting 
brightness values only in darker areas in this way, brightness can be improved in dark areas without changing brightness 
levels in brighter areas. The range of input levels targeted for adjustment may be determined based on sensory test 
of image output results. For example, where the range of possible brightness values is 0 -255, a target range of 180 
35 or less may be set for brightness value adjustment. 

G. Other Embodiments of Backlight Decision Process: 

G1 . Embodiment 2 of Backlight Decision Process: 

40 

[0094] Figs. 14(a) and 14(b) illustrate the backlight decision process In this embodiment. IMG14 in Fig. 14(a) is the 
same as IMG12 in Fig. 12(a). In this embodiment, it is judged that backlighting is present when the following two 
conditions are met. 

[0095] Condition (a1): There was no illumination from a supplemental light source when the image data was gener- 
45 ated. 

Condition (b1): Average brightness value BAave in the subject area is equal to or less than a value obtained by 
multiplying an average brightness value for the entire image by a coefficient k. 

[0096] Condition (b1 ) is the same as that in Embodiment 1 . Decisions as to condition (a1 ) are made on the basis of 
the parameter value for flash information included in image generation record information Gl. In order to improve 

50 brightness of a selected subject when image data is generated under backlit conditions, illumination from a supple- 
mental light source, such as a flash, is sometimes used. It is preferable that no backlight adjustment processing is 
performed where illumination has been provided by a supplemental light source. In this embodiment, if flash information 
includes an operation result of Flash off, condition (a) is determined to have been met. Also, if a reflected light sensing 
mechanism is present but reflected light was not detected, as described hereinabove, condition (a) is determined to 

55 have been met even if the flash operation result is Flash on. Condition (a) is also determined to have been met when 
the parameter value for flash information has been set to "No flash function". When condition (a1) is met in this way, 
that is, where illumination has not been provided by a supplemental light source, the decision is executed based on 
condition (b1 ) just as in Embodiment 1 described previously; and where backlighting is decided to be present, backlight 
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adjustment processing is performed. If, on the other hand, condition (a1) is not met, that is, illumination has been 
provided by a supplemental light source, it is judged that no backlighting is present, and backlight adjustment processing 
is not performed. By making the decision based on flash information in this manner, images in which illumination has 
been provided by a supplemental light source may be excluded from backlight adjustment processing. Additionally, by 
5 making a decision with condition (a1 ) using only flash information and condition (bt ) using both subject area information 
and image data GD, images in which the intended subject is dark may be selected more appropriately as targets for 
backlight adjustment processing. 

G2. Embodiment 3 of Backlight Decision Process: 

10 

[0097] Figs. 15(a) and 15(b) illustrate the backlight decision process in this embodiment. IMG15 in Fig. 15(a), like 
IMG12 in Fig. 12(a), shows an image created under backlit conditions, and has the light source, i.e. the sun S, located 
behind a human figure H, so that human figure H is dark. In image IMG15, a subject area center SAC and a subject 
area S A are established. 

15 [0098] Fig. 15(b) illustrates the backlight decision process in this embodiment. In this embodiment, it is judged that 
backlighting is present when the following three conditions are met. 

[0099] Condition (a1): There was no illumination from a supplemental light source when the image data was gener- 
ated. 

Condition (a2): Subject distance is equal to or less than a threshold value. 
20 Condition (b1): Average brightness value BAave in the subject area is equal to or less than a value obtained by 

multiplying an average brightness value for the entire image by a coefficient k. 

[0100] The difference from the backlight decision process shown in Fig. 14(b) is the additional condition (a2). Where 
conditions (a1) and (b1 ) are met, a decision that backlighting is present is made just as in Embodiment 2 shown in Fig. 
14(b). Where condition (a1) is not met, that is, where illumination has been provided by a supplemental light source, 

2s a further decision is made on the basis of condition (a2). The decision regarding condition (a2) is made based on the 
parameter value for subject distance obtained from the image generation record information Gl. If subject distance, 
distance between the image generating device and a subject, is far, illumination provided by a flash may not sufficiently 
improve brightness in some instances. In such cases, it is preferable to carry out backlight adjustment processing even 
if illumination has been provided by a supplemental light source. In this embodiment, condition (a2) is determined to 

30 have been met if subject distance is equal to or less than a threshold value. Where condition (a2) has been met, a 
further decision is made based on condition (b1) just as in Embodiment 1 described earlier, and if it decided that 
backlighting is present, backlight adjustment processing is performed. By executing a decision regarding condition (a2) 
using subject distance in this way, an image in which illumination has been provided by a supplemental light source, 
but whose brightness has not been sufficiently improved thereby, may be selected as a target for backlight adjustment 

35 processing. The threshold value for deciding the size of subject distance may be determined based on sensory test of 
image output results. For example, a threshold value of 2 meters could be used. 

[0101] Condition (a2) based on subject distance can be considered to be based on the quantity of light falling on a 
subject. If subject distance is less than a threshold value, a large quantity of light falls on the subject, improving its 
brightness. On the other hand, where subject distance is equal to or greater than a threshold value, the quantity of 

*o light is not sufficient to improve subject brightness in some instances. That is, in condition (a2), the threshold value is 
the criterion for determining the magnitude of the quantity of light falling on a subject. Thus, by adjusting this threshold 
value on the basis of other parameter values that can alter the quantity of light falling on a subject, more appropriate 
backlight decisions can be made. For example, an arrangement whereby threshold value increases in association with 
higher flash intensity in image generation record information Gl can be used. By so doing, backlight decisions can be 

45 made appropriately on the basis of flash intensity, i.e., the quantity of light on a subject from a supplemental light source. 
An arrangement whereby threshold value decreases in association with higher aperture value (F number) is also pos- 
sible. A larger F number means less light entering the image generating device. In other words, a larger F number 
means a smaller apparent quantity of light on a subject. Thus, by reducing the threshold value at larger F numbers, 
backlight decisions can be made more appropriately. An arrangement whereby threshold value increases in association 

50 with higher ISO speed rating is also possible. ISO speed rating is an index of the sensitivity of an optical circuit, and 
a larger value means a greater apparent quantity of light entering the image generating device, i.e., a greater apparent 
quantity of light on a subject. Thus, by increasing the threshold value at higher ISO speed ratings, backlight decisions 
can be made more appropriately. 

55 G3, Embodiment 4 of Backlight Decision Process: 

[0102] Figs. 16(a) and 16(b) illustrate the backlight decision process in this embodiment. IMG16 in Fig. 14(a) is the 
same as IMG12 in Fig. 12(a). In this embodiment, it is judged that backlighting is present when the following two 
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conditions are met. 

[0103] Condition (a3): The light source type is an outdoor type. 

Condition (b1): Average brightness value BAave in the subject area is equal to or less than a value obtained by 
multiplying an average brightness value for the entire image by a coefficient k. 

5 [0104] Condition (b1 ) Is the same as that in Embodiment 1 . Decisions as to condition (a3) are made on the basis of 
light source information acquired from image generation record information Gl. The sun is usually the light source when 
a subject is outdoors. Since the position of the sun cannot be set as desired in some instances, generation of image 
data under backlit conditions may be unavoidable. Further, since sun light is intense, subjects under backlit conditions 
tend to be very dark. On the other hand, where a subject is indoors, artificial illumination is usually the light source. 

10 Since the position an artificial light source can usually be set as desired, it is relatively easy to avoid backlighting during 
shooting. In this embodiment, where the light source is determined to be of outdoor type, for example, daylight, clear 
sky, cloudy, or shade, condition (a3) is determined to be met. Where condition (a3) based on light source information 
is met, i.e., where the subject is outdoors, a decision based on condition (b1) is then made in the same way as in 
Embodiment 1 ; and where backlighting is determined to be present, backlight adjustment processing is performed. On 

15 the other hand, where the light source is determined to be of indoor type, for example, either fluorescent or tungsten, 
condition (a3) is determined not to be met, and backlight adjustment processing is not performed. By executing deci- 
sions on the basis of light source information in this way, an image whose subject is outdoors may be selected as a 
target for backlight adjustment processing. Additionally, by making a decision with condition (a3) using only the light 
source information and with condition (b1) using both subject area information and image data GD, images in which 

20 the intended subject is dark may be selected more appropriately as targets for backlight adjustment processing. 

[0105] By using some suitable combination of decision conditions based on image generation record information Gl, 
for example, conditions (a1) -(a3) in the preceding embodiments, more appropriate decisions can be made. For ex- 
ample, in this embodiment, an image determined to not meet condition (a3) may be subject additionally to the decision 
shown in Fig. 14(b) or Fig. 15(b). By so doing, appropriate decisions can be made based on flash information even 

25 for images that have been shot outdoors. 

G4. Embodiment 5 of Backlight Decision Process: 

[0106] Figs. 17(a)-17(c) illustrate a weight W distribution for use in calculating average brightness value in this em- 

30 bodiment. Fig. 17(b) shows an X direction weight W distribution taken along line B-B in the image IMG of Fig. 17(a); 
and Fig. 17(c) shows a Y direction weight W distribution taken along line C-C in the image IMG of Fig. 17(a) In this . 
embodiment, a weighted average value for brightness value is calculated using a weight W distribution that weights 
more heavily pixels closer to the subject area center SAC of image IMG, and the output target decision process is 
performed using this weighted average. 

35 [0107] Figs. 18(a) and 18(b) illustrate the backlight decision process in this embodiment. Image IMG18in Fig. 18(a) 
is the same as image IMG12in Fig. 12(a). Below image IMG18is shown an X direction distribution of weight W, shown 
in Fig. 1 7(b). The Y direction distribution is not shown. Weighted average brightness value obtained using this weight 
W is greater in a backlit image which, like image IMG18, is darker in proximity to the center of the subject area. 
[0108] Fig. 18(b) shows the backlight decision process of this embodiment. In this embodiment, it is judged that 

to backlighting is present when the following condition is met. 

[0109] Condition (b2): Weighted average brightness value BAWave in the subject area is equal to or less than a 
value obtained by multiplying an average brightness value for the entire image Bave by a coefficient k. 
[0110] Specifically, a decision that backlighting is present is made when brightness around the subject area center 
is lower than the overall image brightness. A value determined based on sensory test of image output results may be 

45 used as coefficient k. For example, a vaiue of 0.6 may be used as a predetermined coefficient k. The decision as to 
condition (b2) differs from that for decision condition (b1) described previously in that the subject area center is used 
instead of the subject area. Thus, a decision can be made even for an image in which no subject area has been 
established. The decision as to condition (b2) is made by analyzing the image data GD while assigning greater weight 
to the subject area center, i.e., the subject position, and smaller weight to positions further away from the subject area 

so center, and using the weighted average BAWave obtained as the result of the analysis. By making decisions using 
subject area information in this way, images in which the subject at an intended location is dark can be selected ap- 
propriately as targets for backlight adjustment processing. 

[0111] The overall average Bave can be calculated using another weight distribution having smaller magnitude for 
pixels close to the subject area center. Such a weight distribution can be obtained by subtracting the above weight 
55 from a constant. By so doing, brightness in proximity to the subject area center, i.e. subject location brightness, and 
brightness in the surrounding area may be compared more precisely. 

[0112] By using a suitable combination of decision condition (b2) with other decision conditions based on image 
generation record information Gl, for example, conditions (a1) -(a3) in the preceding embodiments, more appropriate 
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decisions can be made. For example, in the preceding embodiments, this decision condition (b2) could be used instead 
of decision condition (b1). 

G5. Embodiment 5 of Backlight Decision Process: 

5 

[0113] Figs. 19(a) and 19(b) illustrate the backlight decision process in this embodiment. IMG19 in Fig. 19(a), like 
IMG 1 2 in Fig. 1 2(a) : shows an image in backlit conditions. In image IMG1 9 are shown a bright area BA whose brightness 
value is equal to or greater than a brightness threshold value YHth, and a dark area DA whose brightness value is 
equal to or less than a darkness threshold value YLth. In this way, in a backlit image, pixels of high brightness value 
10 and pixels of low brightness value are numerous. 

[0114] Fig. 19(b) shows the backlight decision process in this embodiment. In this embodiment, it is judged that 
backlighting is present when the following conditions are met. 

[0115] Condition (a1): There was no illumination from a supplemental light source when the image data was gener- 
ated. 

15 Condition (b3): The proportion Bratio of pixels whose brightness value is equal to or greater than brightness 

threshold value YHth is equal to or greater than a threshold value Bth (bright pixel proportion threshold value Bth), and 
the proportion Dratio of pixels whose brightness value is equal to or greater than darkness threshold value YLth is 
equal to or greater than another threshold value Dth (dark pixel proportion threshold value Dth). 
[0116] Condition (a1) is the same as that in Embodiment 2 described previously. In contrast to the aforementioned 

20 condition (b1) and condition (b2), the decision regarding condition (b3) is made by analyzing image data GD only. In 
an image generated under backlit conditions, both bright areas and dark areas are present, and thus the proportion of 
bright pixels and the proportion of dark pixels tend to be high. By means of a decision based on condition (b3), such 
an image may be selected as a target for backlight adjustment processing. The threshold values YHth, YLth, Bth, and 
Dth may be determined based on sensory test of image output results. For example, where the range of possible 

25 brightness values is 0 -255, a brightness threshold value YHth of 200 and a darkness threshold value YLth of 50 would 
be acceptable. Bright pixel proportion threshold value Bth may be set to 20% of total pixels, and dark pixel proportion 
threshold value Dth to 30% of total pixels. 

[01 17] The decision regarding condition (b3) in this embodiment can be made without using subject area information. 
Thus, backlight decisions can be made even where image generation record information Gl does not include subject 

30 area information. In this embodiment, backlight decisions are made using not only condition (b3) decided on by analysis 
of image data GD, but condition (a1) decided on using flash information, i.e. image generation record information Gl, 
so more appropriate backlight decision results for images can be obtained. Various conditions other than condition 
(a1) could be used as decision conditions based on image generation record information Gl. For example, more ap- 
propriate decisions could be made through suitable combination of conditions (a1 ) -(a3) in the preceding embodiments. 

35 in any event, more appropriate decisions could be made by combining decisions made through analysis of image data 
GD only, with decisions made using image generation record information Gl. Decisions based on image data GD only 
are not limited to condition (b3); other conditions enabling a decision to be made as to whether backlit conditions have 
resulted in dark areas are acceptable. For example, pixels in an image could be divided into a number of blocks, and 
brightness values for the blocks may be used as average brightness value for the block. A decision that backlighting 

40 is present may be made where the proportion of bright blocks whose brightness value is equal to or greater than 
threshold value YHth and the proportion of dark blocks whose brightness value is equal to or less than threshold value 
YLth are each equal to or greater than their predetermined threshold values, 

H. Other Embodiments of Backlight Adjustment Processing: 

45 

H1 . Embodiment 2 of Backlight Adjustment Processing: 

[0118] Figs. 20(a)-20(c) illustrate the backlight decision process in this embodiment. The two images IMG20a, 
IMG20b in Figs. 20(a) and 20(b), like the image IMG12 shown in Fig. 12 (a), show images under backlit conditions. In 

so the two images IMG20a, IMG20b, a human figure H appears dark due to being backlit. The human figure H in image 
IMG20b appears darker than that in image IMG20a. Thus, pixels with low brightness value are more numerous in 
image IMG20b than in image IMG20a. As an index of the degree of brightness of a dark area in such an image, it is 
possible to use, for example, a pixel brightness value whose brightness value magnitude rank in the image is a pre- 
determined rank (20% of total pixels, for example) counting from the lowest brightness value (hereinafter termed "dark 

55 area brightness value"). For example, when the image includes 100 pixels, brightness of the pixel whose brightness 
is the 20th from the lowest is used as the dark area brightness value Bm. Such a dark area brightness value Bm will 
be small in a backlit image like image IMG20a, and even smaller in a darker image like IMG20b. 
[0119] Fig. 20(c) shows the relationship of brightness value input level Yin to brightness value output level Yout in 
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backlight adjustment processing of this embodiment. In graph G1 , the amount of increase in brightness value is adjusted 
such that where input level Yin is equal to the dark area brightness value Bm1 , output level Yout is raised to a prede- 
termined target value BO higher than the original value. Hereinafter, the input level serving as the benchmark for raising 
the brightness value is termed "the brightness adjustment benchmark level." Output levels Yout corresponding to other 
5 input levels Yin are interpolated using a spline function. By adjusting brightness values using this graph G1 , brightness 
in dark areas can be improved. 

[0120] Graphs G2 and G3 show input/output characteristics used for backlight adjustment processing with greater 
intensity than graph G1 . Here, "backlight adjustment processing intensity" refers to the extent of increase in brightness 
value, particularly of increase in brightness value in dark areas. Graph G2 is used when a brightness adjustment 

10 benchmark level, i.e., dark area brightness value, Bm2 is smaller than Bm1. The amount of increase in brightness 
value is adjusted such that where input level Yin is equal to Bm2, output level Yout is raised to the predetermined target 
value BO. Using graph G2, more intense backlight adjustment processing than with graph G1 can be carried out. Graph 
G3 is used when a brightness adjustment benchmark level Bm3 is even smaller than Bm2. Using graph G3, more 
intense backlight adjustment processing than with graph G2 can be carried out. In this embodiment, backlight adjust- 

15 ment processing intensity can be determined through analysis of image data GD alone. 

[0121] By adjusting backlight adjustment processing intensity based on an index that indicates the degree of bright- 
ness in dark areas of an image in this way, backlight adjustment processing can be performed with intensity that is 
appropriate to the brightness of a particular dark area. That is, the lower the degree of brightness in a dark area (i.e. 
the darker it is), the more intense can be the backlight adjustment processing performed on it. Various other indexes 

20 other than dark area brightness value can be used as indexes for the degree of brightness in dark areas. For example, 
the proportion of dark pixels having brightness values below a predetermined threshold value may be used. In this 
case, the greater the proportion of dark pixels, the more intense will be the backlight adjustment processing. In any 
case, by adjusting backlight adjustment processing to a more intense level the lower the degree of brightness in a dark 
area (i.e. the darker it is), backlight adjustment processing can be performed with intensity appropriate to the brightness 

25 of a particular dark area. As the predetermined target value BO, a value determined through sensory test of image 
output results may be used. For example, where the range of possible brightness values is 0 -255, the median value 
of 128 may be used as the target value BO. 
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H2. Embodiment 3 of Backlight Adjustment Processing: 



[0122] Figs. 21(a)-21(c) illustrate the backlight decision process in this embodiment. The two images JMG21a, 
IMG21b shown in Figs. 21(a) and 21(b), unlike the images IMG20a, IMG20b shown in Figs. 20(a) and 20(b), include 
information for a subject area center SAC and a subject area SA. A human figure H that is dark due to being backlit is 
shown in each of the two images IMG21a, IMG21b. The human figure H in image IMG21b appears darker than that 

35 in image IMG21a.Thus, pixels with low brightness value are more numerous in image IMG21bthan in image IMG21a. 
An index indicating the degree of brightness in such a backlit image may be determined using subject area information. 
As an index indicating brightness of a dark area, there may be used the maximum brightness value BAmax within the 
subject area. Such a maximum brightness value BAmax will be small in a backlit image like image IMG21a, and even 
smaller in a darker backlit image like image IMG21b. 

40 [0123] Fig. 21(c) shows the relationship of brightness value input level Yin to brightness value output level Yout in 
backlight adjustment processing of this embodiment. A difference from the example shown in Fig. 20(c) is that the 
maximum brightness value in the subject area is used as the brightness adjustment benchmark level. In graph G1 , the 
amount of increase in brightness value is adjusted such that where input level Yin is equal to BAmaxI (maximum 
brightness value in the subject area), brightness value output level Yout is raised to a predetermined target value BO. 

45 Graph G2 depicts an input-output function used where the brightness adjustment benchmark level is BAmax2, which 
is smaller than BAmaxI ; and graph G3 depicts an input-output function used where the brightness adjustment bench- 
mark level is BAmax3, which is even smaller than BAmax2. In this way, since the intensity of backlight adjustment 
processing is greater the smaller the maximum brightness value in a subject area, brightness in a subject area can be 
improved appropriately. It should be noted that, in this embodiment, backlight adjustment processing intensity is de- 

50 termined using both image data GD and subject area information, that is, image data GD and image generation record 
information Gl. In other words, the intensity of backlight adjustment processing is determined based on the results of 
analyzing image data GD while assigning zero weight to the area outside the subject area. That is, the image data GD 
is analyzed using a weight distribution of different magnitude in the subject location (subject area center) than in the 
area outside the subject area. 

55 [0124] By adjusting processing intensity based on an index that indicates the brightness in a dark area, backlight 
adjustment processing can be performed with the intensity appropriate to the brightness of a subject area. Various 
values other than maximum brightness value in the subject area can be used as indexes of brightness in a dark area. 
For example, the average brightness value for the subject may be used as the index. By so doing, more appropriate 
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adjustment of backlight adjustment processing intensity may be accomplished even where a subject area includes a 
bright area. A weighted average brightness value calculated using the weight W distribution shown in Figs. 17(a)-17 
(c) can also be used as an index. In this case, average brightness value is calculated using the subject area center 
rather than the subject area, so more appropriate adjustment of processing intensity may be accomplished based on 
5 brightness values in proximity to the subject, even for an image in which no subject area has been established. In any 
case, by adjusting backlight adjustment processing to a more intense level the lower the degree of brightness in a 
subject area (i.e. the darker it is), backlight adjustment processing can be performed at intensity appropriate to the 
brightness of a particular subject area. 

10 H3. Variant Examples of Backlight Adjustment Processing: 

[0125] In the embodiments described above, the target value BO for raising brightness value is a predetermined 
value, but it can instead be a value adjusted based on an index that indicates the degree of brightness in a dark area 
within an image. For example, the target value BO may be increased as the dark area brightness value decreases. By 

15 so doing, the intensity of backlight adjustment processing can be increased with reference to darker brightness of a 
dark area : so that brightness can be improved. Target value BO may also be increased as the maximum brightness 
value within a subject area decreases. By so doing, the intensity of backlight adjustment processing can be increased 
with reference to darker brightness of a subject area, so that brightness can be improved. Alternatively, a number of 
brightness value adjustment benchmark levels may be devised, and the target value for raising brightness value set 

20 on a level-by-level basis. For example, brightness value adjustment benchmark levels may be set to correspond to 
average brightness value and maximum brightness value in a subject area, respectively, and the amount of increase 
in brightness value may be adjusted accordingly. By so doing, finer adjustment of brightness value becomes possible. 
Or, the aforementioned brightness value adjustment benchmark level may be a predetermined value, and instead, the 
target value BO for raising brightness value adjusted based on an index of the dark area. In any case, by adjusting 

25 backlight adjustment processing to a more intense level the lower the degree of brightness in a dark area (i.e. the 
darker it is) produced by backlighting, backlight adjustment processing can be performed at intensity appropriate to 
the brightness of a particular subject area. 

I. Arrangement of Image Output System Employing Image Data Processing Device: 

30 

[0126] Fig. 22 is an explanatory diagram showing an example of an image output system in which an image data 
processing device may be implemented by way of an embodiment of the present invention. Image output system 10B 
comprises a digital still camera 12 as an image generating device for generating image files; a computer PC for exe- 
cuting image quality adjustment processing based on an image file; and a printer 20B as an image output device for 

35 outputting images. Computer PC functions as an image data processing device. As the image output device, a CRT 
display, LCD display, or other monitor 14B, and a projector may be used instead of printer 20B. In the following de- 
scription, it is assumed that printer 20B is used as the output section. This embodiment differs from the image output 
system embodiment described previously (Fig. 1) in that the image processing device including a judging section and 
an image quality adjuster is constructed separately from the output section. Combination of the computer PC serving 

40 as the image data processing device and the printer serving as the output section can be termed an "image output 
device" in the broad sense. 

[0127] An image file created in digital still camera 12 is transferred to computer PC via a cable CV, or by directly 
inserting into computer PC a memory card MC having the image file stored thereon. Computer PC executes image 
quality adjustment processing of the image data based on the read out image file. The image data produced by image 

45 quality adjustment processing is transferred to printer 20B via a cable CV, and output by printer 20B. 

[0128] Computer PC comprises a CPU 1 50 for executing a program that realizes the aforementioned image quality 
adjustment processing; RAM 151 for temporarily storing results of operations by CPU 150; and a hard disk drive (HDD) 
152 for storing data needed for image quality adjustment processing : such as an image quality adjustment processing 
program, lookup tables, and aperture value tables. CPU 1 50, RAM 151, and HDD 1 52 as a whole function as a judging 

so section and image quality adjuster. Computer PC further comprises a memory card slot 153 for installing a memory 
card MC; and an input/output terminal 154 for connecting a connector cable from digital still camera 12. 
[0129] An image file GF generated by a digital still camera 12 is supplied to computer PC via a cable or via a memory 
card MC. When an image data processing application program, either an image retouching application or a printer 
driver, is started up by user control, CPU 150 executes an image processing routine (Fig. 9) to process the read in 

55 image file GF Alternatively, the image data processing application program may be set to start up automatically when 
a memory card MC is inserted into memory card slot 153, or when connection of a digital still camera 12 to input/output 
terminal 154 via a cable is detected. 

[0130] Image data processed by CPU 150, rather than being output in Step S250 of the image processing routine 
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(Fig. 9), is instead transferred to an image output device, for example, printer 20B, whereupon the image output device 
receiving the image data executes image output. 

[0131 ] In this embodiment, image processing is carried out using a judging section and image quality adjuster realized 
by computer PC, whereby an output device not equipped with a judging section and image quality adjuster may be 
used. Where the output device is equipped with a judging section and image quality adjuster, image data may be 
transferred directly to the output device without being processed by the computer PC, and the image is processed by 
the judging section and image quality adjuster of the output device. 

[0132] As described in the embodiments hereinabove, image quality of an image containing a dark area produced 
by backlighting may be adjusted automatically, thereby readily producing high quality image output. 

J. Variant Examples: 

J1. Variant Example 1: 

[01 33] An arrangement whereby one or more conditions for a backlight adjustment processing decision are selected 
from among a number of decision conditions according to information included in the image generation record infor- 
mation Gl is also possible. For example, where image generation record information Gl includes flash information and 
subject distance information, the decisions can be made based on conditions (a1) (a2) and (a3); and where image 
generation record information Gl includes flash information and subject area information, the decisions can be made 
based on conditions (at ) and (b1 ). By selecting conditions according to information included in image generation record 
information Gl, more appropriate decisions can be made. 

J2. Variant Example 2: 

[0134] As information relating to the distance between a subject and the image generating device, employed in 
condition (a2), another distance information indicating a distance range can be used instead. For example, the decision 
regarding condition (a2) can be made using subject distance range information set by selecting from among three 
distance ranges, macro (0-1 m), close-view (1 -3 m), or distant-view (3 m or over). In this case, the decision regarding 
condition (a2) can be made by pre-determined typical distance for each of the distance ranges, and comparing the 
typical distance with a threshold value for condition (a2). As a typical distance, for example, for a distance range having 
an upper limit value and a lower limit value established for distance, the intermediate value could be used; or for a 
distance range having only an upper limit value or a lower limit value established, the upper limit value or lower limit 
value could be used. 

J3. Variant Example 3: 

[0135] Where image data is generated with a supplemental light source and the image generating device situated 
at separate locations, it is preferable to make the decision regarding condition (a2) based on information relating to 
the distance between the supplemental light source and the image generating device, rather than subject distance. By 
so doing, images containing dark areas due to a large distance between supplemental light source and subject may 
be selected as targets for backlight adjustment processing. 

J4. Variant Example 4: 

[0136] Where an image file GF does not include an image data gamma value or color space information, the color 
space conversion process of the image processing routine shown in Fig. 1 0 (Step S320 and Step S330) can be omitted. 
Fig. 23 is a flow chart showing an image processing routine from which the color space conversion process has been 
omitted. Image data acquired in Step S500 is converted from image data based on a YCbCr color space to data based 
on an RGB color space in Step S51 0. Next, in Step S520, automatic image quality adjustment processing is executed 
using the image data obtained in Step S510. Next, in Step S530, a CYMK conversion process and halftone process 
for printing are executed. 

J5. Variant Example 5: 

[0137] In the preceding embodiments, automatic image quality adjustment processing is executed after executing 
color space conversion, but instead color space conversion can be executed after executing automatic image quality 
adjustment processing. For example, image processing can be executed according to the flowchart shown in Fig. 24. 
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J6. Variant Example 6: 

[0138] In the preceding embodiments, a printer is used as the output section, but an output section other than a 
printer could be used instead. Fig. 25 is a flowchart showing a processing routine for image processing based on image 

5 generation record Information where a CRT is used as the output section. In contrast to the flowchart in Fig. 1 0 which 
depicts the use of a printer as the image output section , the CYMK conversion process and halftone process for printing 
are omitted. Also, since a CRT can represent an RGB color space of image data obtained by executing a matrix 
operation (S), the color space conversion process is omitted as well. Where image data based on an RGB color space 
obtained in Step S610 includes data outside the defined area of the RGB color space, the out-of-defined-area data is 

10 clipped, and then step S620 is executed. Where the color space utilizable by an image output section is different from 
an RGB color space, a color conversion process to a color space utilizable by the image output section is executed in 
a manner analogous to executing a CMYK color conversion process where a printer is used, and the resultant image 
is output by the image output section. 

15 J7. Variant Example 7: 

[01 39] In the flowchart of Fig. 1 1 , the chroma adjusting process S430 and noise elimination process S440 performed 
after backlight adjustment processing S420 may be omitted. This produces a simpler arrangement for image quality 
processing. 

20 

J8. Variant Example 8: 

[0140] In the preceding embodiments, the description was made taking an Exif format file as a specific example of 
the image file GF, but the format of the image file pertaining to the present invention is not limited to this. That is, any 
25 image file containing image data generated by an image generating device, and image generation record information 
Gl describing conditions at the time of generation of image data is acceptable. With such a file, image quality of image 
data generated by an image generating device can be appropriately adjusted automatically and output from an output 
device. 

30 J9. Variant Example 9: 

[0141] Values of matrices S, N _1 , and M in the equations are merely exemplary, and can be modified appropriately 
depending on colorspace on which the image file is based, color space utilizable by an image output section, or the like. 

35 J 1 0 . Variant Example 1 0 : 

[0142] In the preceding embodiments, the description was made using a digital still camera 1 2 as the image gener- 
ating device, but image files could be generated using a different image generating device, such as a scanner, and 
digital video camera. 

40 

J11 . Variant Example 11 : 

[0143] In the preceding embodiments, the description has been made for the example of a case where image data 
GD and image generation record information Gl are contained in the same image file GF, but image data GD and image 

45 generation record information Gl need not necessarily be stored within the same file. That is, it is sufficient for image 
data GD and image generation record information Gl to be associated with each other; for example, it would be ac- 
ceptable to generate associating data that associates image data GD with image generation record information Gl; 
store one or several sets of image data and image generation record information Gl in independent files; and refer to 
the associated image generation record information Gl when processing the image data GD. This is because, in this 

so case, although the image data GD and image generation record information Gl are stored in separate files, at the point 
in time of image processing which utilizes the image generation record information Gl, the image data GD and image 
generation record information Gl are in inseparably linked, and thus function substantially the same as if they were 
stored in the same file. That is, the term image file GF in the present embodiment includes those of a form wherein 
image data GD and image generation record information Gl are associated, at least at the point in time that image 

55 processing takes place. Also included are motion video files stored on optical disk media such as CD-ROM, CD-R, 
DVD-ROM, and DVD-RAM. 

[0144] Although the present invention has been described and illustrated in detail, it is clearly understood that the 
same is by way of illustration and example only and is not to be taken by way of limitation, the spirit and scope of the 
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present invention being limited only by the terms of the appended claims. 



Claims 

1 . An image processing device for processing an image using image data generated by an image generating device, 
and image generation record information that is associated with the image data and that includes operation infor- 
mation for the image generating device at the time that the image data is generated, the image processing device 
comprising: 

a judging section configured to execute a backlight decision as to whether or not to execute backlight adjust- 
ment processing, based on both the image generation record information and the image data; and 
an image quality adjuster that, when it is decided to execute the backlight adjustment processing, executes 
backlight adjustment processing to increase brightness value of at least some pixels in the image data. 

2. An image processing device according to claim 1 , wherein 

when the image generation record information includes subject position information indicating a position of 
a subject in the image, the judging section uses the subject position information in executing the backlight decision. 

3. An image processing device according to claim 2, wherein 
the judging section analyses the image data with a weight distribution that has different magnitudes at the 

subject position and other positions, and execute the backlight decision according to the analysis result. 

An image processing device according to claim 1 or 3, wherein 

when the image generation record information includes flash information of a supplemental light source at 
the time of generation of the image data, the judging section decides based on the flash information whether 
illumination with light by the supplemental light source has been performed at the time of generation of the image 
data, and uses a result of this decision in executing the backlight decision. 

An image processing device according to claim 4, wherein 

the judging section, based on the flash information, is able to identify one among available operation results 
of the supplemental light source at the time of generation of the image data, and 

the judging section executes the backlight decision based on brightness values of the image data when the 
operation result is one of the following results: 

(i) no supplemental light source is provided; 

(ii) the supplemental light source is not fired; and 

(iii) the supplemental light source is fired, and reflected light is detected. 

6. An image processing device according to claim 5, wherein 

the' image generation record information further includes information relating to a distance between the sub- 
ject of the image data and the image generating device at the time of generation of the image data, and 
the judging section performs: 

comparing the subject distance to a predetermined threshold value when the supplemental light source oper- 
ation result is not any of the results (i), (ii) and (iii); 

executing the backlight decision using the brightness values of the image data when a decision that the subject 
distance is equal to or greater than the predetermined threshold value; and 

deciding not to execute the backlight adjustment processing when a decision that the subject distance is less 
than the predetermined threshold value. 

7. An image processing device according to any one of claims 1 to 6, wherein 

when the image generation record information includes information relating to location of the subject of the 
image data, the judging section decides whether the subject location is an outdoor location, and executes the 
backlight decision depending on the decision result. 

8. An image processing device according claim 7, wherein 

when a decision that the subject location is an outdoor location is made, the judging section executes the 
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backlight decision using brightness values of the image data. 



9. An image processing device according to any one of claims 1 to 8, wherein 

the image quality adjuster determines intensity of the backlight adjustment processing based on both the 
5 image generation record information and the image data. 

10. An image processing device according to claim 9, wherein 

when the image generation record information includes subject position information indicating a position of 
a subject in the image, the image quality adjuster analyses the image data with a weight distribution that has 
10 different magnitudes at the subject position and other positions, and determines intensity of the backlight adjust- 

ment processing according to the analysis result. 

11. An image output device for outputting an image using image data generated by an image generating device, and 
image generation record information that is associated with the image data and that includes operation information 

15 for the image generating device at the time that the image data is generated, the image output device comprises: 

an image processing device according to any one of claims 1 to 10; and 

an output section for outputting an image according to the image quality-adjusted image data. 

20 12. A method of processing an image using image data generated by an image generating device, and image gener- 
ation record information that is associated with the image data and that includes operation information for the 
image generating device at the time that the image data is generated, the method comprising the steps of: 

(a) executing a backlight decision as to whether or not to execute backlight adjustment processing, based on 
25 both the image generation record information and the image data; and 

(b) when it is decided to execute the backlight adjustment processing, executing backlight adjustment process- 
ing to increase brightness value of at least some pixels in the image data. 

13. A method according to claim 12, wherein 

30 when the image generation record information includes subject position information indicating a position of 

a subject in the image, the backlight decision is made using the subject position information. 

14. A method according to claim 13, wherein 

the step (a) includes analyzing the image data with a weight distribution that has different magnitudes at the 
35 subject position and other positions, and executing the backlight decision according to the analysis result. 

15. A method according to claim 12 or 14, wherein 

when the image generation record information includes flash information of a supplemental light source at 
the time of generation of the image data, the step (a) includes deciding based on the flash information whether 
*o illumination with light by the supplemental light source has been performed at the time of generation of the image 

data, and executing the backlight decision using a result of this decision. 

16. A method according to claim 15, wherein 

the step (a) includes, based on the flash information, identifying one among available operation results of 
^5 the supplemental light source at the time of generation of the image data, and 

the step (a) includes executing the backlight decision based on brightness values of the image data when 
the operation result is one of the following results: 

(i) no supplemental light source is provided; 
so (jj) the supplemental light source is not fired; and 

(iii) the supplemental light source is fired, and reflected light is detected. 

17. A method according to claim 16, wherein 

the image generation record information further includes information relating to a distance between the sub- 
55 ject of the image data and the image generating device at the time of generation of the image data, and 

the step (a) includes: 

comparing the subject distance to a predetermined threshold value when the supplemental light source oper- 
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ation result is not any of the results (ii) and (iii); 

executing the backlight decision using the brightness values of the image data when a decision that the subject 
distance is equal to or greater than the predetermined threshold value; and 

deciding not to execute the backlight adjustment processing when a decision that the subject distance is less 
s than the predetermined threshold value. 

18. A method according to any one of claims 12 to 17, wherein 

when the image generation record information includes information relating to location of the subject of the 
image data, the step (a) includes deciding whether the subject location is an outdoor location, and executing the 
io backlight decision depending on the decision result. 

19. A method according claim 18, wherein 

when a decision that the subject location is an outdoor location is made, the step (a) includes executing the 
backlight decision using brightness values of the image data. 

15 

20. A method according to any one of claims 12 to 19, wherein 

the step (a) includes determining intensity of the backlight adjustment processing based on both the image 
generation record information and the image data. 

20 21 . A method according to claim 20, wherein 

when the image generation record information includes subject position information indicating a position of 
a subject in the image, the step (b) includes analyzing the image data with a weight distribution that has different 
magnitudes at the subject position and other positions, and determining intensity of the backlight adjustment 
processing according to the analysis result. 

25 

22. A method of outputting an image using image data generated by an image generating device, and image generation 
record information that is associated with the image data and that includes operation information for the image 
generating device at the time that the image data is generated, the method comprising: 

30 the steps (a) and (b) according to any one of claims 12 to 21 ; and 

outputting an image according to the image quality-adjusted image data. 

23. A computer program product comprising: 

35 a computer readable medium; and 

a computer program stored on the computer readable medium, the computer program including a program 
for causing a computer to implement a method according to any one of claims 12 to 21 . 
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Fig.5 
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Fig.24 
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